Simplified HDFS Architecture with Blockchain Distribution of Metadata

Big data storage becomes one of the great challenges due to the rapid growth of huge volume, variety, velocity and veracity of data from various sources like social sites, Internet of Things, mobile users and others. These data cannot be processed by the traditional database systems. Hadoop is a distributed and massively parallel processing system for big data whereby the storage is based on the distributed file system called HadoopDistributed File System (HDFS). HDFS is organized as a collection of Data Nodes which stores data as blocks and the information regarding the data is kept as metadata which is stored in an expensive, reliable hardware known as Name Node, which serves as a master server. The existing HDFS architecture suffers from single point of failure due to the existence of single Name Node, where the metadata creation is being done. Metadata is the whole information regarding the distribution of data blocks, replica management and block size information.This paper proposed a simplified architecture for big data storage which eliminates the concept of master node called Name Node with the functionalities of the Name Node being distributed using blockchain technology. Metadata creation and blockchain placement was implemented and tested in a cluster of nodes. The code for metadata creation in python runs successfully for the proposed architecture. Theimplementation of blockchain in the proposed methodology results in low metadata access delay and thereby improves the execution time.